Patient monitoring apparatus

ABSTRACT

A monitoring system for a person in a building includes a plurality of wireless bases positioned in the building; a body mounted temperature sensor; a body mounted heart signal sensor; and a wearable device coupled to the temperature sensor and to the heart signal sensor and having a wireless transceiver adapted to communicate with the wireless bases; and an accelerometer.

This application is a continuation of Ser. No. 11/252,279, filed Oct.16, 2005 and which issued on Sep. 2, 2008 as U.S. Pat. No. 7,420,472,the content of which is incorporated by reference.

BACKGROUND

Improvements in living condition and advances in health care haveresulted in a marked prolongation of life expectancy for elderly anddisabled population. These individuals, a growing part of society, aredependent upon the delivery of home health and general care, which hasits own set of challenges and drawbacks. This population needscontinuous general, as well as medical, supervision and care. Forexample, supervision of daily activities such as dressing, personalhygiene, eating and safety as well as supervision of their health statusis necessary. Furthermore, the relief of loneliness and anxiety is amajor, yet unsolved, problem that has to be dealt with.

The creation of retirement facilities and old age homes, as well asother geriatric facilities, provides only a partial solution to theproblems facing the geriatric population. As discussed in U.S. Pat. Nos.5,544,649 and 6,433,690, the notion of ambulatory (home environment)patient care is gaining increased popularity and importance. Asdiscussed in the '649 patent, the number of old aged people receivinghome care services under Medicare has shown a 13% annual growth rate andhas tripled in 10 years (1978-1988) from 769,000 to 2.59 million. Thisshift in patient care from the “sheltered” institutional milieu to thepatient's home, work place, or recreational environment is driven inpart by cost and in part to a new care concept that prefers keeping theaged and the disabled in their own natural environment for as long aspossible.

Typically, home care is carried out either by the patient's family or bynonprofessional help. The monitoring equipment at home care facilitiesis usually minimal or nonexistent, and the patient has to be transportedto the doctor's office or other diagnostic facility to allow properevaluation and treatment. Patient follow-up is done by means of homevisits of nurses which are of sporadic nature, time consuming andgenerally very expensive. A visiting nurse can perform about 5-6 homevisits per day. The visits have to be short and can usually not becarried out on a daily basis. Moreover, a visiting nurse programprovides no facilities for continuous monitoring of the patient and thusno care, except in fortuitous circumstances, in times of emergency. Theremainder of day after the visiting nurse has left is often a period ofisolation and loneliness for the elderly patient.

In outpatient care, health-threatening falls are an importantepidemiological problem in this growing segment of the population.Studies indicate that approximately two thirds of accidents in people 65years of age or older, and a large percentage of deaths from injuries,are due to falls. Approximately 1.6 million hip fracture injuriesworldwide in 1990 were due to falls, and that this number will increase6.26% by 2050, with the highest incidences recorded in Northern Europeand North America. In the elderly, 90% of hip fractures happen at age 70and older, and 90% are due to falls. The falls are usually due (80%) topathological balance and gait disorders and not to overwhelming externalforce (i.e., being pushed over by some force). More than 50% of elderlypersons suffer from arthritis and/or orthopedic impairments, whichfrequently leads to falls. Specifically prone to falls are womenexperiencing a higher percentage of arthritis-related structural bonechanges. It is estimated that approximately 5% of falls result infracture and 1% of all falls are hip fractures. The percentages varyslightly in different geographical regions (e.g., Japan, Scandinavia),but the consensus of the available research is that the falls are asignificant epidemiological problem in the growing elderly population.

The physiological and medical results of a fall range from grazes, cuts,sprains, bruising, fractures, broken bones, torn ligaments, permanentinjuries and death. The most common fall-related injuries arelacerations and fractures, the most common area of fracture being thehip and wrist. Damage to areas such as the hip and legs can result inpermanent damage and increased likelihood or further falls and injuries.

Falls in the elderly can produce social and mental as well asphysiological repercussions. Anguish and reduced confidence in mobilitycan complicate problems and even increase the probability of falls andseverity of fall-related injuries from lack of exercise and tensing ofmuscles.

Among older people in the U.S. (age 65+) there are approximately 750,000falls per year requiring hospitalization due to either bone fracturing(approx. 480,000 cases) or hip fracturing (approx. 270,000 cases). Theresult of such injuries is an average hospital stay between 2 and 8days. Assuming the average cost of $1,000 per hospital day, a total costof falls in the elderly for the health care industry can be estimated atthree billion dollars per year. This figure is likely to increase as theolder age segment of the population increases.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary embodiment for monitoring a person.

FIG. 2A illustrates a process for determining three dimensional (3D)detection.

FIG. 2B shows an exemplary calibration sheet.

FIG. 3 illustrates a process for detecting falls.

FIG. 4 illustrates a process for detecting facial expressions.

FIG. 5 illustrates a smart cane that works with the embodiment of FIG.1.

FIG. 6 shows an exemplary wrist-watch based assistance device.

SUMMARY

In one aspect, a monitoring system for a person in a building includes aplurality of wireless bases positioned in the building; a body mountedtemperature sensor; a body mounted heart signal sensor; and a wearabledevice coupled to the temperature sensor and to the heart signal sensorand having a wireless transceiver adapted to communicate with thewireless bases; and an accelerometer.

In another aspect, a monitoring system for a person includes one or morewireless bases; and a wearable device such as a wristwatch having awireless transceiver adapted to communicate with the one or morewireless bases; and an accelerometer to detect a dangerous condition andto generate a warning when the dangerous condition is detected.

Implementations of the above aspect may include one or more of thefollowing. The wristwatch determines position based on triangulation.The wristwatch determines position based on RF signal strength and RFsignal angle. A switch detects a confirmatory signal from the person.The confirmatory signal includes a head movement, a hand movement, or amouth movement. The confirmatory signal includes the person's voice. Aprocessor in the system executes computer readable code to transmit ahelp request to a remote computer. The code can encrypt or scramble datafor privacy. The processor can execute voice over IP (VOIP) code toallow a user and a remote person to audibly communicate with each other.The voice communication system can include Zigbee VOIP or Bluetooth VOIPor 802.XX VOIP. The remote person can be a doctor, a nurse, a medicalassistant, or a caregiver. The system includes code to store and analyzepatient information. The patient information includes medicine takinghabits, eating and drinking habits, sleeping habits, or excise habits. Apatient interface is provided on a user computer for accessinginformation and the patient interface includes in one implementation atouch screen; voice-activated text reading; and one touch telephonedialing. The processor can execute code to store and analyze informationrelating to the person's ambulation. A global positioning system (GPS)receiver can be used to detect movement and where the person falls. Thesystem can include code to map the person's location onto an area forviewing. The system can include one or more cameras positioned tocapture three dimensional (3D) video of the patient; and a servercoupled to the one or more cameras, the server executing code to detecta dangerous condition for the patient based on the 3D video and allow aremote third party to view images of the patient when the dangerouscondition is detected.

In another aspect, a monitoring system for a person includes one or morewireless bases; and a cellular telephone having a wireless transceiveradapted to communicate with the one or more wireless bases; and anaccelerometer to detect a dangerous condition and to generate a warningwhen the dangerous condition is detected.

In yet another aspect, a monitoring system includes one or more camerasto determine a three dimensional (3D) model of a person; means to detecta dangerous condition based on the 3D model; and means to generate awarning when the dangerous condition is detected.

In another aspect, a method to detect a dangerous condition for aninfant includes placing a pad with one or more sensors in the infant'sdiaper; collecting infant vital parameters; processing the vitalparameter to detect SIDS onset; and generating a warning.

Advantages of the invention may include one or more of the following.The system provides timely assistance and enables elderly and disabledindividuals to live relatively independent lives. The system monitorsphysical activity patterns, detects the occurrence of falls, andrecognizes body motion patterns leading to falls. Continuous monitoringof patients is done in an accurate, convenient, unobtrusive, private andsocially acceptable manner since a computer monitors the images andhuman involvement is allowed only under pre-designated events. Thepatient's privacy is preserved since human access to videos of thepatient is restricted: the system only allows human viewing underemergency or other highly controlled conditions designated in advance bythe user. When the patient is healthy, people cannot view the patient'svideo without the patient's consent. Only when the patient's safety isthreatened would the system provide patient information to authorizedmedical providers to assist the patient. When an emergency occurs,images of the patient and related medical data can be compiled and sentto paramedics or hospital for proper preparation for pick up and checkinto emergency room.

The system allows certain designated people such as a family member, afriend, or a neighbor to informally check on the well-being of thepatient. The system is also effective in containing the spiraling costof healthcare and outpatient care as a treatment modality by providingremote diagnostic capability so that a remote healthcare provider (suchas a doctor, nurse, therapist or caregiver) can visually communicatewith the patient in performing remote diagnosis. The system allowsskilled doctors, nurses, physical therapists, and other scarce resourcesto assist patients in a highly efficient manner since they can do themajority of their functions remotely.

Additionally, a sudden change of activity (or inactivity) can indicate aproblem. The remote healthcare provider may receive alerts over theInternet or urgent notifications over the phone in case of such suddenaccident indicating changes. Reports of health/activity indicators andthe overall well being of the individual can be compiled for the remotehealthcare provider. Feedback reports can be sent to monitored subjects,their designated informal caregiver and their remote healthcareprovider. Feedback to the individual can encourage the individual toremain active. The content of the report may be tailored to the targetrecipient's needs, and can present the information in a formatunderstandable by an elder person unfamiliar with computers, via anappealing patient interface. The remote healthcare provider will haveaccess to the health and well-being status of their patients withoutbeing intrusive, having to call or visit to get such informationinterrogatively. Additionally, remote healthcare provider can receive areport on the health of the monitored subjects that will help themevaluate these individuals better during the short routine check upvisits. For example, the system can perform patient behavior analysissuch as eating/drinking/smoke habits and medication compliance, amongothers.

The patient's home equipment is simple to use and modular to allow forthe accommodation of the monitoring device to the specific needs of eachpatient. Moreover, the system is simple to install.

DESCRIPTION

FIG. 1 shows an exemplary home monitoring system. In this system, aplurality of monitoring cameras 10 are placed in various predeterminedpositions in a home of a patient 30. The cameras 10 can be wired orwireless. For example, the cameras can communicate over infrared linksor over radio links conforming to the 802.11X (e.g. 802.11A, 802.11B,802.11G) standard or the Bluetooth standard to a server 20. The server20 stores images and videos of patient's ambulation pattern.

The patient 30 may wear one or more sensors 40, for example devices forsensing ECG, EKG, blood pressure, sugar level, among others. In oneembodiment, the sensors 40 are mounted on the patient's wrist (such as awristwatch sensor) and other convenient anatomical locations. Exemplarysensors 40 include standard medical diagnostics for detecting the body'selectrical signals emanating from muscles (EMG and EOG) and brain (EEG)and cardiovascular system (ECG). Leg sensors can include piezoelectricaccelerometers designed to give qualitative assessment of limb movement.Additionally, thoracic and abdominal bands used to measure expansion andcontraction of the thorax and abdomen respectively. A small sensor canbe mounted on the subject's finger in order to detect blood-oxygenlevels and pulse rate. Additionally, a microphone can be attached tothroat and used in sleep diagnostic recordings for detecting breathingand other noise. One or more position sensors can be used for detectingorientation of body (lying on left side, right side or back) duringsleep diagnostic recordings. Each of sensors 40 can individuallytransmit data to the server 20 using wired or wireless transmission.Alternatively, all sensors 40 can be fed through a common bus into asingle transceiver for wired or wireless transmission. The transmissioncan be done using a magnetic medium such as a floppy disk or a flashmemory card, or can be done using infrared or radio network link, amongothers. The sensor 40 can also include a global position system (GPS)receiver that relays the position and ambulatory patterns of the patientto the server 20 for mobility tracking.

In one embodiment, the sensors 40 for monitoring vital signs areenclosed in a wrist-watch sized case supported on a wrist band. Thesensors can be attached to the back of the case. For example, in oneembodiment, Cygnus' AutoSensor (Redwood City, Calif.) is used as aglucose sensor. A low electric current pulls glucose through the skin.Glucose is accumulated in two gel collection discs in the AutoSensor.The AutoSensor measures the glucose and a reading is displayed by thewatch.

In another embodiment, EKG/ECG contact points are positioned on the backof the wrist-watch case. In yet another embodiment that providescontinuous, beat-to-beat wrist arterial pulse rate measurements, apressure sensor is housed in a casing with a ‘free-floating’ plunger asthe sensor applanates the radial artery. A strap provides a constantforce for effective applanation and ensuring the position of the sensorhousing to remain constant after any wrist movements. The change in theelectrical signals due to change in pressure is detected as a result ofthe piezoresistive nature of the sensor are then analyzed to arrive atvarious arterial pressure, systolic pressure, diastolic pressure, timeindices, and other blood pressure parameters.

The case may be of a number of variations of shape but can beconveniently made a rectangular, approaching a box-like configuration.The wrist-band can be an expansion band or a wristwatch strap ofplastic, leather or woven material. The wrist-band further contains anantenna for transmitting or receiving radio frequency signals. Thewristband and the antenna inside the band are mechanically coupled tothe top and bottom sides of the wrist-watch housing. Further, theantenna is electrically coupled to a radio frequency transmitter andreceiver for wireless communications with another computer or anotheruser. Although a wrist-band is disclosed, a number of substitutes may beused, including a belt, a ring holder, a brace, or a bracelet, amongother suitable substitutes known to one skilled in the art. The housingcontains the processor and associated peripherals to provide thehuman-machine interface. A display is located on the front section ofthe housing. A speaker, a microphone, and a plurality of push-buttonswitches and are also located on the front section of housing. Aninfrared LED transmitter and an infrared LED receiver are positioned onthe right side of housing to enable the watch to communicate withanother computer using infrared transmission.

In another embodiment, the sensors 40 are mounted on the patient'sclothing. For example, sensors can be woven into a single-piece garment(an undershirt) on a weaving machine. A plastic optical fiber can beintegrated into the structure during the fabric production processwithout any discontinuities at the armhole or the seams. Aninterconnection technology transmits information from (and to) sensorsmounted at any location on the body thus creating a flexible “bus”structure. T-Connectors—similar to “button clips” used in clothing—areattached to the fibers that serve as a data bus to carry the informationfrom the sensors (e.g., EKG sensors) on the body. The sensors will pluginto these connectors and at the other end similar T-Connectors will beused to transmit the information to monitoring equipment or personalstatus monitor. Since shapes and sizes of humans will be different,sensors can be positioned on the right locations for all patients andwithout any constraints being imposed by the clothing. Moreover, theclothing can be laundered without any damage to the sensors themselves.In addition to the fiber optic and specialty fibers that serve assensors and data bus to carry sensory information from the wearer to themonitoring devices, sensors for monitoring the respiration rate can beintegrated into the structure.

In another embodiment, instead of being mounted on the patient, thesensors can be mounted on fixed surfaces such as walls or tables, forexample. One such sensor is a motion detector. Another sensor is aproximity sensor. The fixed sensors can operate alone or in conjunctionwith the cameras 10. In one embodiment where the motion detectoroperates with the cameras 10, the motion detector can be used to triggercamera recording. Thus, as long as motion is sensed, images from thecameras 10 are not saved. However, when motion is not detected, theimages are stored and an alarm may be generated. In another embodimentwhere the motion detector operates stand alone, when no motion issensed, the system generates an alarm.

The server 20 also executes one or more software modules to analyze datafrom the patient. A module 50 monitors the patient's vital signs such asECG/EKG and generates warnings should problems occur. In this module,vital signs can be collected and communicated to the server 20 usingwired or wireless transmitters. In one embodiment, the server 20 feedsthe data to a statistical analyzer such as a neural network which hasbeen trained to flag potentially dangerous conditions. The neuralnetwork can be a back-propagation neural network, for example. In thisembodiment, the statistical analyzer is trained with training data wherecertain signals are determined to be undesirable for the patient, givenhis age, weight, and physical limitations, among others. For example,the patient's glucose level should be within a well established range,and any value outside of this range is flagged by the statisticalanalyzer as a dangerous condition. As used herein, the dangerouscondition can be specified as an event or a pattern that can causephysiological or psychological damage to the patient. Moreover,interactions between different vital signals can be accounted for sothat the statistical analyzer can take into consideration instanceswhere individually the vital signs are acceptable, but in certaincombinations, the vital signs can indicate potentially dangerousconditions. Once trained, the data received by the server 20 can beappropriately scaled and processed by the statistical analyzer. Inaddition to statistical analyzers, the server 20 can process vital signsusing rule-based inference engines, fuzzy logic, as well as conventionalif-then logic. Additionally, the server can process vital signs usingHidden Markov Models (HMMs), dynamic time warping, or template matching,among others.

Through various software modules, the system reads video sequence andgenerates a 3D anatomy file out of the sequence. The proper bone andmuscle scene structure are created for head and face. A based profilestock phase shape will be created by this scene structure. Every scenewill then be normalized to a standardized viewport.

A module 52 monitors the patient ambulatory pattern and generateswarnings should the patient's patterns indicate that the patient hasfallen or is likely to fall. 3D detection is used to monitor thepatient's ambulation. In the 3D detection process, by putting 3 or moreknown coordinate objects in a scene, camera origin, view direction andup vector can be calculated and the 3D space that each camera views canbe defined.

In one embodiment with two or more cameras, camera parameters (e.g.field of view) are preset to fixed numbers. Each pixel from each cameramaps to a cone space. The system identifies one or more 3D featurepoints (such as a birthmark or an identifiable body landmark) on thepatient. The 3D feature point can be detected by identifying the samepoint from two or more different angles. By determining the intersectionfor the two or more cones, the system determines the position of thefeature point. The above process can be extended to certain featurecurves and surfaces, e.g. straight lines, arcs; flat surfaces,cylindrical surfaces. Thus, the system can detect curves if a featurecurve is known as a straight line or arc. Additionally, the system candetect surfaces if a feature surface is known as a flat or cylindricalsurface. The further the patient is from the camera, the lower theaccuracy of the feature point determination. Also, the presence of morecameras would lead to more correlation data for increased accuracy infeature point determination. When correlated feature points, curves andsurfaces are detected, the remaining surfaces are detected by texturematching and shading changes. Predetermined constraints are appliedbased on silhouette curves from different views. A different constraintcan be applied when one part of the patient is occluded by anotherobject. Further, as the system knows what basic organic shape it isdetecting, the basic profile can be applied and adjusted in the process.

In a single camera embodiment, the 3D feature point (e.g. a birth mark)can be detected if the system can identify the same point from twoframes. The relative motion from the two frames should be small butdetectable. Other features curves and surfaces will be detectedcorrespondingly, but can be tessellated or sampled to generate morefeature points. A transformation matrix is calculated between a set offeature points from the first frame to a set of feature points from thesecond frame. When correlated feature points, curves and surfaces aredetected, the rest of the surfaces will be detected by texture matchingand shading changes.

Each camera exists in a sphere coordinate system where the sphere origin(0,0,0) is defined as the position of the camera. The system detectstheta and phi for each observed object, but not the radius or size ofthe object. The radius is approximated by detecting the size of knownobjects and scaling the size of known objects to the object whose sizeis to be determined. For example, to detect the position of a ball thatis 10 cm in radius, the system detects the ball and scales otherfeatures based on the known ball size. For human, features that areknown in advance include head size and leg length, among others. Surfacetexture can also be detected, but the light and shade information fromdifferent camera views is removed. In either single or multiple cameraembodiments, depending on frame rate and picture resolution, certainundetected areas such as holes can exist. For example, if the patientyawns, the patient's mouth can appear as a hole in an image. For 3Dmodeling purposes, the hole can be filled by blending neighborhoodsurfaces. The blended surfaces are behind the visible line.

In one embodiment shown in FIG. 2A, each camera is calibrated before 3Ddetection is done. Pseudo-code for one implementation of a cameracalibration process is as follows:

-   -   Place a calibration sheet with known dots at a known distance        (e.g. 1 meter), and perpendicular to a camera view.    -   Take snap shot of the sheet, and correlate the position of the        dots to the camera image:        -   Dot1(x,y,1)            pixel(x,y)    -   Place a different calibration sheet that contains known dots at        another different known distance (e.g. 2 meters), and        perpendicular to camera view.    -   Take another snapshot of the sheet, and correlate the position        of the dots to the camera image:        -   Dot2(x,y,2)            pixel(x,y)    -   Smooth the dots and pixels to minimize digitization errors. By        smoothing the map using a global map function, step errors will        be eliminated and each pixel will be mapped to a cone space.    -   For each pixel, draw a line from Dot1(x,y,z) to Dot2(x, y, z)        defining a cone center where the camera can view.    -   One smoothing method is to apply a weighted filter for Dot1 and        Dot2. A weight filter can be used. In one example, the following        exemplary filter is applied.

$\quad\begin{matrix}1 & 2 & 1 \\2 & 4 & 2 \\1 & 2 & 1\end{matrix}$

-   -   Assuming Dot1_Left refers to the value of the dot on the left        side of Dot1 and Dot1_Right refers to the value of the dot to        the right of Dot1 and Dot1_Upper refers to the dot above Dot1,        for example, the resulting smoothed Dot1 value is as follows:        1/16*(Dot1*4+Dot1_Left*2+Dot1_Right*2+Dot1_Upper*2+Dot1_Down*2+Dot1_UpperLeft+Dot1_UpperRight+Dot1_LowerLeft+Dot1_LowerRight)    -   Similarly, the resulting smoothed Dot2 value is as follows:        1/16*(Dot2*4+Dot2_Left*2+Dot2_Right*2+Dot2_Upper*2+Dot2_Down*2+Dot2_UpperLeft+Dot2_UpperRight+Dot2_LowerLeft+Dot2_LowerRight)

In another smoothing method, features from Dot1 sheet are mapped to asub pixel level and features of Dot2 sheet are mapped to a sub pixellevel and smooth them. To illustrate, Dot1 dot center (5, 5, 1) aremapped to pixel (1.05, 2.86), and Dot2 dot center (10, 10, 2) are mappedto pixel (1.15, 2.76). A predetermined correlation function is thenapplied.

FIG. 2B shows an exemplary calibration sheet having a plurality of dots.In this embodiment, the dots can be circular dots and square dots whichare interleaved among each others. The dots should be placed relativelyclose to each other and each dot size should not be too large, so we canhave as many dots as possible in one snapshot. However, the dots shouldnot be placed too close to each other and the dot size should not be toosmall, so they are not identifiable.

A module 54 monitors patient activity and generates a warning if thepatient has fallen. In one implementation, the system detects the speedof center of mass movement. If the center of mass movement is zero for apredetermined period, the patient is either sleeping or unconscious. Thesystem then attempts to signal the patient and receive confirmatorysignals indicating that the patient is conscious. If patient does notconfirm, then the system generates an alarm. For example, if the patienthas fallen, the system would generate an alarm signal that can be sentto friends, relatives or neighbors of the patient. Alternatively, athird party such as a call center can monitor the alarm signal. Besidesmonitoring for falls, the system performs video analysis of the patient.For example, during a particular day, the system can determine theamount of time for exercise, sleep, and entertainment, among others. Thenetwork of sensors in a patient's home can recognize ordinarypatterns—such as eating, sleeping, and greeting visitors—and to alertcaretakers to out-of-the-ordinary ones—such as prolonged inactivity orabsence. For instance, if the patient goes into the bathroom thendisappears off the sensor for 13 minutes and don't show up anywhere elsein the house, the system infers that patient had taken a bath or ashower. However, if a person falls and remains motionless for apredetermined period, the system would record the event and notify adesignated person to get assistance.

A fall detection process (shown in FIG. 3) performs the followingoperations:

-   -   Find floor space area    -   Define camera view background 3D scene    -   Calculate patient's key features    -   Detect fall

In one implementation, pseudo-code for determining the floor space areais as follows:

-   -   1. Sample the camera view space by M by N, e.g. M=1000, N=500.    -   2. Calculate all sample points the 3D coordinates in room        coordinate system; where Z axis is pointing up. Refer to the 3D        detection for how to calculate 3D positions.    -   3. Find the lowest Z value point (Zmin)    -   4. Find all points whose Z values are less than Zmin+Ztol; where        Ztol is a user adjustable value, e.g. 2 inches.    -   5. If rooms have different elevation levels, then excluding the        lowest Z floor points, repeat step 2, 3 and 4 while keeping the        lowest Z is within Ztol2 of previous Z. In this example Ztol2=2        feet, which means the floor level difference should be within 2        feet.    -   6. Detect stairs by finding approximate same flat area but        within equal Z differences between them.    -   7. Optionally, additional information from the user can be used        to define floor space more accurately, especially in single        camera system where the coordinates are less accurate, e.g.:        -   a. Import the CAD file from constructors' blue prints.        -   b. Pick regions from the camera space to define the floor,            then use software to calculate its room coordinates.        -   c. User software to find all flat surfaces, e.g. floors,            counter tops, then user pick the ones, which are actually            floors and/or stairs.

In the implementation, pseudo-code for determining the camera viewbackground 3D scene is as follows:

-   -   1. With the same sample points, calculate x, y coordinates and        the Z depth and calculate 3D positions.    -   2. Determine background scene using one the following methods,        among others:        -   a. When there is nobody in the room.        -   b. Retrieve and update the previous calculated background            scene.        -   c. Continuous updating every sample point when the furthest            Z value was found, that is the background value.

In the implementation, pseudo-code for determining key features of thepatient is as follows:

-   -   1. Foreground objects can be extracted by comparing each sample        point's Z value to the background scene point's Z value, if it        is smaller, then it is on the foreground.    -   2. In normal condition, the feet/shoe can be detected by finding        the lowest Z point clouds close the floor in room space, its        color will be extracted.    -   3. In normal condition, the hair/hat can be detected by finding        the highest Z point clouds close the floor in room space, its        color will be extracted.    -   4. The rest of the features can be determined by searching from        either head or toe. E.g., hat, hair, face, eye, mouth, ear,        earring, neck, lipstick, moustache, jacket, limbs, belt, ring,        hand, etc.    -   5. The key dimension of features will be determined by        retrieving the historic stored data or recalculated, e.g., head        size, mouth width, arm length, leg length, waist, etc.    -   6. In abnormal conditions, features can be detected by detect        individual features then correlated them to different body        parts. E.g., if patient's skin is black, we can hardly get a        yellow or white face, by detecting eye and nose, we know which        part is the face, then we can detect other characteristics.

To detect fall, the pseudo-code for the embodiment is as follows:

-   -   1. The fall has to be detected in almost real time by tracking        movements of key features very quickly. E.g. if patient has        black hair/face, track the center of the black blob will know        roughly where his head move to.    -   2. Then the center of mass will be tracked, center of mass is        usually around belly button area, so the belt or borderline        between upper and lower body closed will be good indications.    -   3. Patient's fall always coupled with rapid deceleration of        center of mass. Software can adjust this threshold based on        patient age, height and physical conditions.    -   4. Then if the fall is accidental and patient has difficult to        get up, one or more of following will happen:        -   a. Patient will move very slowly to find support object to            get up.        -   b. Patient will wave hand to camera ask for help. To detect            this condition, the patient hand has to be detected first by            finding a blob of points with his skin color. Hand motion            can be tracked by calculate the motion of the center of the            points, if it swings left and right, it means patient is            waving to camera.        -   c. Patient is unconscious, motionless. To detect this            condition, extract the foreground object, calculate its            motion vectors, if it is within certain tolerance, it means            patient is not moving. In the mean time, test how long it            last, if it past a user defined time threshold, it means            patient is in great danger.

In one embodiment for fall detection, the system determines a patientfall-down as when the patient's knee, butt or hand is on the floor. Thefall action is defined a quick deceleration of center of mass, which isaround belly button area. An accidental fall action is defined when thepatient falls down with limited movement for a predetermined period.

The system monitors the patients' fall relative to a floor. In oneembodiment, the plan of the floor is specified in advance by thepatient. Alternatively, the system can automatically determine the floorlayout by examining the movement of the patient's feet and estimated thesurfaces touched by the feet as the floor.

The system detects a patient fall by detecting a center of mass of anexemplary feature. Thus, the software can monitor the center of one ormore objects, for example the head and toe, the patient's belt, thebottom line of the shirt, or the top line of the pants.

The detection of the fall can be adjusted based on two thresholds:

a. Speed of deceleration of the center of mass.

b. The amount of time that the patient lies motionless on the floorafter the fall.

In one example, once a stroke occurs, the system detects a slow motionof patient as the patient rests or a quick motion as the patientcollapses. By adjust the sensitivity threshold, the system detectswhether a patient is uncomfortable and ready to rest or collapse.

If the center of mass movement ceases to move for a predeterminedperiod, the system can generate the warning. In another embodiment,before generating the warning, the system can request the patient toconfirm that he or she does not need assistance. The confirmation can bein the form of a button that the user can press to override the warning.Alternatively, the confirmation can be in the form of a single utterancethat is then detected by a speech recognizer.

In another embodiment, the confirmatory signal is a patient gesture. Thepatient can nod his or her head to request help and can shake the headto cancel the help request. Alternatively, the patient can use aplurality of hand gestures to signal to the server 20 the actions thatthe patient desires.

By adding other detecting mechanism such as sweat detection, the systemcan know whether patient is uncomfortable or not. Other items that canbe monitored include chest movement (frequency and amplitude) and restlength when the patient sits still in one area, among others.

Besides monitoring for falls, the system performs video analysis of thepatient. For example, during a particular day, the system can determinethe amount of time for exercise, sleep, entertainment, among others. Thenetwork of sensors in a patient's home can recognize ordinarypatterns—such as eating, sleeping, and greeting visitors—and to alertcaretakers to out-of-the-ordinary ones—such as prolonged inactivity orabsence. For instance, if the patient goes into the bathroom thendisappears off the camera 10 view for a predetermined period and doesnot show up anywhere else in the house, the system infers that patienthad taken a bath or a shower. However, if a person falls and remainsmotionless for a predetermined period, the system would record the eventand notify a designated person to get assistance.

In one embodiment, changes in the patient's skin color can be detectedby measuring the current light environment, properly calibrating colorspace between two photos, and then determining global color changebetween two states. Thus, when the patient's face turn red, based on theredness, a severity level warning is generated.

In another embodiment, changes in the patient's face are detected byanalyzing a texture distortion in the images. If the patient perspiresheavily, the texture will show small glisters, make-up smudges, orsweat/tear drippings. Another example is, when long stretched face willbe detected as texture distortion. Agony will show certain wrinkletexture patterns, among others.

The system can also utilize high light changes. Thus, when the patientsweats or changes facial appearance, different high light areas areshown, glisters reflect light and pop up geometry generates more highlight areas.

A module 62 analyzes facial changes such as facial asymmetries. Thechange will be detected by superimpose a newly acquired 3D anatomystructure to a historical (normal) 3D anatomy structure to detectface/eye sagging or excess stretch of facial muscles.

In one embodiment, the system determines a set of base 3D shapes, whichare a set of shapes which can represent extremes of certain facialeffects, e.g. frown, open mouth, smiling, among others. The rest of the3D face shape can be generated by blending/interpolating these baseshapes by applied different weight to each base shapes.

The base 3D shape can be captured using 1) a 3D camera such as camerasfrom Steinbichler, Genex Technology, Minolta 3D, Olympus 3D or 2) one ormore 2D camera with preset camera field of view (FOV) parameters. Tomake it more accurate, one or more special markers can be placed onpatient's face. For example, a known dimension square stick can beplaced on the forehead for camera calibration purposes.

Using the above 3D detection method, facial shapes are then extracted.The proper features (e.g. a wrinkle) will be detected and attached toeach base shape. These features can be animated or blended by changingthe weight of different shape(s). The proper features change can bedetected and determine what type of facial shape it will be.

Next, the system super-imposes two 3D facial shapes (historical ornormal facial shapes and current facial shapes). By matching featuresand geometry of changing areas on the face, closely blended shapes canbe matched and facial shape change detection can be performed. Byoverlaying the two shapes, the abnormal facial change such as saggingeyes or mouth can be detected.

The above processes are used to determine paralysis of specific regionsof the face or disorders in the peripheral or central nervous system(trigeminal paralysis; CVA, among others). The software also detectseyelid positions for evidence of ptosis (incomplete opening of one orboth eyelids) as a sign of innervation problems (CVA; Horner syndrome,for example). The software also checks eye movements for pathologicalconditions, mainly of neurological origin are reflected in aberrationsin eye movement. Pupil reaction is also checked for abnormal reaction ofthe pupil to light (pupil gets smaller the stronger the light) mayindicate various pathological conditions mainly of the nervous system.In patients treated for glaucoma pupillary status and motion pattern maybe important to the follow-up of adequate treatment. The software alsochecks for asymmetry in tongue movement, which is usually indicative ofneurological problems. Another check is neck veins: Engorgement of theneck veins may be an indication of heart failure or obstruction ofnormal blood flow from the head and upper extremities to the heart. Thesoftware also analyzes the face, which is usually a mirror of theemotional state of the observed subject. Fear, joy, anger, apathy areonly some of the emotions that can be readily detected, facialexpressions of emotions are relatively uniform regardless of age, sex,race, etc. This relative uniformity allows for the creation of computerprograms attempting to automatically diagnose people's emotional states.

Speech recognition is performed to determine a change in the form ofspeech (slurred speech, difficulties in the formation of words, forexample) may indicated neurological problems, such an observation canalso indicate some outward effects of various drugs or toxic agents.

In one embodiment shown in FIG. 4, a facial expression analysis processperforms the following operations:

-   -   Find floor space area    -   Define camera view background 3D scene    -   Calculate patient's key features    -   Extract facial objects    -   Detect facial orientation    -   Detect facial expression

The first three steps are already discussed above. The patient's keyfeatures provide information on the location of the face, and once theface area has been determined, other features can be detected bydetecting relative position to each other and special characteristics ofthe features:

-   -   Eye: pupil can be detected by applying Chamfer matching        algorithm, by using stock pupil objects.    -   Hair: located on the top of the head, using previous stored hair        color to locate the hair point clouds.    -   Birthmarks, wrinkles and tattoos: pre store all these features        then use Chamfer matching to locate them.    -   Nose: nose bridge and nose holes usually show special        characteristics for detection, sometime depend on the view        angle, is side view, special silhouette will be shown.    -   Eye browse, Lips and Moustache: All these features have special        colors, e.g. red lipstick; and base shape, e.g. patient has no        expression with mouth closed. Software will locate these        features by color matching, then try to deform the base shape        based on expression, and match shape with expression, we can        detect objects and expression at the same time.    -   Teeth, earring, necklace: All these features can be detected by        color and style, which will give extra information.

In one implementation, pseudo-code for detecting facial orientation isas follows:

-   -   Detect forehead area    -   Use the previously determined features and superimpose them on        the base face model to detect a patient face orientation.

Depends on where patient is facing, for a side facing view, silhouetteedges will provide unique view information because there is a one to onecorrespondent between the view and silhouette shape.

Once the patient's face has been aligned to the right view, exemplarypseudo code to detect facial expression is as follows:

-   -   1. Detect shape change. The shape can be match by superimpose        different expression shapes to current shape, and judge by        minimum discrepancy. E.g. wide mouth open.    -   2. Detect occlusion. Sometime the expression can be detected by        occlusal of another objects, e.g., teeth show up means mouth is        open.    -   3. Detect texture map change. The expression can relate to        certain texture changes, if patient smile, certain wrinkles        patents will show up.    -   4. Detect highlight change. The expression can relate to certain        high light changes, if patient sweats or cries, different        highlight area will show up.

A module 80 communicates with a third party such as the policedepartment, a security monitoring center, or a call center. The module80 operates with a POTS telephone and can use a broadband medium such asDSL or cable network if available. The module 80 requires that at leastthe telephone is available as a lifeline support. In this embodiment,duplex sound transmission is done using the POTS telephone network. Thebroadband network, if available, is optional for high resolution videoand other advanced services transmission.

During operation, the module 80 checks whether broadband network isavailable. If broadband network is available, the module 80 allows highresolution video, among others, to be broadcasted directly from theserver 20 to the third party or indirectly from the server 20 to theremote server 200 to the third party. In parallel, the module 80 allowssound to be transmitted using the telephone circuit. In this manner,high resolution video can be transmitted since sound data is separatelysent through the POTS network.

If broadband network is not available, the system relies on the POTStelephone network for transmission of voice and images. In this system,one or more images are compressed for burst transmission, and at therequest of the third party or the remote server 200, the telephone'ssound system is placed on hold for a brief period to allow transmissionof images over the POTS network. In this manner, existing POTS lifelinetelephone can be used to monitor patients. The resolution and quantityof images are selectable by the third party. Thus, using only thelifeline as a communication medium, the person monitoring the patientcan elect to only listen, to view one high resolution image with duplextelephone voice transmission, to view a few low resolution images, toview a compressed stream of low resolution video with digitized voice,among others.

During installation or while no live person in the scene, each camerawill capture its own environment objects and store it as backgroundimages, the software then detect the live person in the scene, changesof the live person, so only the portion of live person will be send tothe local server, other compression techniques will be applied, e.g.send changing file, balanced video streaming based on change.

The local server will control and schedule how the video/picture will besend, e.g. when the camera is view an empty room, no pictures will besent, the local server will also determine which camera is at the rightview, and request only the corresponding video be sent. The local serverwill determine which feature it is interested in looking at, e.g. faceand request only that portion be sent.

With predetermined background images and local server controlledstreaming, the system will enable higher resolution and more camerasystem by using narrower bandwidth.

Through this module, a police officer, a security agent, or a healthcareagent such as a physician at a remote location can engage, ininteractive visual communication with the patient. The patient's healthdata or audio-visual signal can be remotely accessed. The patient alsohas access to a video transmission of the third party. Should thepatient experience health symptoms requiring intervention and immediatecare, the health care practitioner at the central station may summonhelp from an emergency services provider. The emergency servicesprovider may send an ambulance, fire department personnel, familymember, or other emergency personnel to the patient's remote location.The emergency services provider may, perhaps, be an ambulance facility,a police station, the local fire department, or any suitable supportfacility.

Communication between the patient's remote location and the centralstation can be initiated by a variety of techniques. One method is bymanually or automatically placing a call on the telephone to thepatient's home or from the patient's home to the central station.

Alternatively, the system can ask a confirmatory question to the patientthrough text to speech software. The patient can be orally instructed bythe health practitioner to conduct specific physical activities such asspecific arm movements, walking, bending, among others. The examinationbegins during the initial conversation with the monitored subject. Anychanges in the spontaneous gestures of the body, arms and hands duringspeech as well as the fulfillment of nonspecific tasks are importantsigns of possible pathological events. The monitoring person caninstruct the monitored subject to perform a series of simple tasks whichcan be used for diagnosis of neurological abnormalities. Theseobservations may yield early indicators of the onset of a disease.

A network 100 such as the Internet receives images from the server 20and passes the data to one or more remote servers 200. The images aretransmitted from the server 20 over a secure communication link such asvirtual private network (VPN) to the remote server(s) 200.

The server 20 collects data from a plurality of cameras and uses the 3Dimages technology to determine if the patient needs help. The system cantransmit video (live or archived) to the friend, relative, neighbor, orcall center for human review. At each viewer site, after a viewerspecifies the correct URL to the client browser computer, a connectionwith the server 20 is established and user identity authenticated usingsuitable password or other security mechanisms. The server 200 thenretrieves the document from its local disk or cache memory storage andtransmits the content over the network. In the typical scenario, theuser of a Web browser requests that a media stream file be downloaded,such as sending, in particular, the URL of a media redirection file froma Web server. The media redirection file (MRF) is a type of specializedHypertext Markup Language (HTML) file that contains instructions for howto locate the multimedia file and in what format the multimedia file isin. The Web server returns the MRF file to the user's browser program.The browser program then reads the MRF file to determine the location ofthe media server containing one or more multimedia content files. Thebrowser then launches the associated media player application programand passes the MRF file to it. The media player reads the MRF file toobtain the information needed to open a connection to a media server,such as a URL, and the required protocol information, depending upon thetype of medial content is in the file. The streaming media content fileis then routed from the media server down to the user.

Next, the transactions between the server 20 and one of the remoteservers 200 are detailed. The server 20 compares one image frame to thenext image frame. If no difference exists, the duplicate frame isdeleted to minimize storage space. If a difference exists, only thedifference information is stored as described in the JPEG standard. Thisoperation effectively compresses video information so that the cameraimages can be transmitted even at telephone modem speed of 64 k or less.More aggressive compression techniques can be used. For example, patientmovements can be clusterized into a group of known motion vectors, andpatient movements can be described using a set of vectors. Only thevector data is saved. During view back, each vector is translated into apicture object which is suitably rasterized. The information can also becompressed as motion information.

Next, the server 20 transmits the compressed video to the remote server200. The server 200 stores and caches the video data so that multipleviewers can view the images at once since the server 200 is connected toa network link such as telephone line modem, cable modem, DSL modem, andATM transceiver, among others.

In one implementation, the servers 200 use RAID-5 striping and paritytechniques to organize data in a fault tolerant and efficient manner.The RAID (Redundant Array of Inexpensive Disks) approach is welldescribed in the literature and has various levels of operation,including RAID-5, and the data organization can achieve data storage ina fault tolerant and load balanced manner. RAID-5 provides that thestored data is spread among three or more disk drives, in a redundantmanner, so that even if one of the disk drives fails, the data stored onthe drive can be recovered in an efficient and error free manner fromthe other storage locations. This method also advantageously makes useof each of the disk drives in relatively equal and substantiallyparallel operations. Accordingly, if one has a six gigabyte clustervolume which spans three disk drives, each disk drive would beresponsible for servicing two gigabytes of the cluster volume. Each twogigabyte drive would be comprised of one-third redundant information, toprovide the redundant, and thus fault tolerant, operation required forthe RAID-5 approach. For additional physical security, the server can bestored in a Fire Safe or other secured box, so there is no chance toerase the recorded data, this is very important for forensic analysis.

The system can also monitor the patient's gait pattern and generatewarnings should the patient's gait patterns indicate that the patient islikely to fall. The system will detect patient skeleton structure,stride and frequency; and based on this information to judge whetherpatient has joint problem, asymmetrical bone structure, among others.The system can store historical gait information, and by overlayingcurrent structure to the historical (normal) gait information, gaitchanges can be detected.

The system also provides a patient interface 90 to assist the patient ineasily accessing information. In one embodiment, the patient interfaceincludes a touch screen; voice-activated text reading; one touchtelephone dialing; and video conferencing. The touch screen has largeicons that are pre-selected to the patient's needs, such as his or herfavorite web sites or application programs. The voice activated textreading allows a user with poor eye-sight to get information from thepatient interface 90. Buttons with pre-designated dialing numbers, orvideo conferencing contact information allow the user to call a friendor a healthcare provider quickly.

In one embodiment, medicine for the patient is tracked using radiofrequency identification (RFID) tags. In this embodiment, each drugcontainer is tracked through an RFID tag that is also a drug label. TheRF tag is an integrated circuit that is coupled with a mini-antenna totransmit data. The circuit contains memory that stores theidentification Code and other pertinent data to be transmitted when thechip is activated or interrogated using radio energy from a reader. Areader consists of an RF antenna, transceiver and a micro-processor. Thetransceiver sends activation signals to and receives identification datafrom the tag. The antenna may be enclosed with the reader or locatedoutside the reader as a separate piece. RFID readers communicatedirectly with the RFID tags and send encrypted usage data over thepatient's network to the server 20 and eventually over the Internet 100.The readers can be built directly into the walls or the cabinet doors.

In one embodiment, capacitively coupled RFID tags are used. Thecapacitive RFID tag includes a silicon microprocessor that can store 96bits of information, including the pharmaceutical manufacturer, drugname, usage instruction and a 40-bit serial number. A conductive carbonink acts as the tag's antenna and is applied to a paper substratethrough conventional printing means. The silicon chip is attached toprinted carbon-ink electrodes on the back of a paper label, creating alow-cost, disposable tag that can be integrated on the drug label. Theinformation stored on the drug labels is written in a Medicine MarkupLanguage (MML), which is based on the eXtensible Markup Language (XML).MML would allow all computers to communicate with any computer system ina similar way that Web servers read Hyper Text Markup Language (HTML),the common language used to create Web pages.

After receiving the medicine container, the patient places the medicinein a medicine cabinet, which is also equipped with a tag reader. Thissmart cabinet then tracks all medicine stored in it. It can track themedicine taken, how often the medicine is restocked and can let thepatient know when a particular medication is about to expire. At thispoint, the server 20 can order these items automatically. The server 20also monitors drug compliance, and if the patient does not remove thebottle to dispense medication as prescribed, the server 20 sends awarning to the healthcare provider.

Due to its awareness of the patient's position, the server 20 canoptionally control a mobility assistance device such as a smart cane.FIG. 5 shows an exemplary robot 1000 for assisting the patient inambulating his or her home. The robot embodiment of FIG. 5 isessentially a smart cane with a camera 1010, a frame 1100 with drivesystems 1110 having stepper motors, wheels, belts and pulleys, mountedto a mounting plate. The robot 1000 also has control module 1200including a processor, memory, camera, display, wireless networking, anddata storage devices. In one embodiment, the control module 1200 is a PCcompatible laptop computer.

The robot 1000 sends video from its camera to the server 20, which inturn coordinates the position of the robot 1000, as determined by thecameras 10 mounted in the home as well as the robot camera. The robotposition, as determined by the server 20, is then transmitted to therobot 1000 for navigation.

In this smart cane embodiment, the frame 1100 has an extended handle1150. The handle 1150 includes handle sensors 1152 mounted thereon todetect the force places on each handle to receive as input the movementdesired by the patient. In one embodiment, the robot 1000 has a controlnavigation system that accepts patient command as well as robotself-guidance command. The mobility is a result of give-and-take betweenthe patient's self-propulsion and the walker's automated reactions.Thus, when the patient moves the handle to the right, the robotdetermines that the patient is interested in turning and actuates thedrive systems 1110 appropriately. However, if the patient is turninginto an obstacle, as determined by the cameras and the server 20, thedrive system provides gentle resistance that tells the patient of animpending collision.

If, for example, a patient does not see a coffee table ahead, the walkerwill detect it, override the patient's steering to avoid it, and therebyprevent a possible fall. Onboard software processes the data from 180degrees of approaching terrain and steers the front wheel towardopenings and away from obstacles.

The control module 1200 executes software that enables the robot 1000 tomove around its environment safely. The software performs localization,mapping, path planning and obstacle avoidance. In one embodiment, imagesfrom a plurality of wall-mounted cameras 10 are transmitted to theserver 20. The server 20 collects images of the robot and triangulatesthe robot position by cross-referencing the images. The information isthen correlated with the image from the robot-mounted camera 1010 andoptical encoders 1011 that count the wheel rotations to calculatetraveled distance for range measurement. In this process, a visual mapof unique “landmarks” created as the robot moves along its path isannotated with the robot's position to indicate the position estimate ofthe landmark. The current image, seen from the robot, is compared withthe images in the database to find matching landmarks. Such matches areused to update the position of the robot according to the relativeposition of the matching landmark. By repeatedly updating the positionof landmarks based on new data, the software incrementally improves themap by calculating more accurate estimates for the position of thelandmarks. An improved map results in more accurate robot positionestimates. Better position estimates contribute to better estimates forthe landmark positions and so on. If the environment changes so muchthat the robot no longer recognizes previous landmarks, the robotautomatically updates the map with new landmarks. Outdated landmarksthat are no longer recognized can easily be deleted from the map bysimply determining if they were seen or matched when expected.

Using the obstacle avoidance algorithm, the robot generates correctivemovements to avoid obstacles not represented in the path planner such asopen/closed doors, furniture, people, and more. The robot rapidlydetects obstacles using its sensors and controls its speed and headingto avoid obstacles.

The hazard avoidance mechanisms provide a reflexive response tohazardous situations to insure the robot's safety and guarantee that itdoes not damage itself or the environment. Mechanisms for hazardavoidance include collision detection using not one but a complementaryset of sensors and techniques. For instance, collision avoidance can beprovided using contact sensing, motor load sensing, and vision. Thecombination of multiple sources for collision detection guarantees safecollision avoidance. Collision detection provides a last resort fornegotiating obstacles in case obstacle avoidance fails to do so in thefirst place, which can be caused by moving objects or software andhardware failures.

If the walker is in motion (as determined by the wheel encoder), theforce applied to the brake pads is inversely proportional to thedistance to obstacles. If the walker is stopped, the brakes should befully applied to provide a stable base on which the patient can rest.When the walker is stopped and the patient wishes to move again, thebrakes should come off slowly to prevent the walker from lurchingforward The walker should mostly follow the patient's commands, as thisis crucial for patient acceptance. For the safety braking and the safetybraking and steering control systems, the control system only influencesthe motion when obstacles or cliffs are near the patient. In otherwords, the walker is, typically, fully patient controlled. For all othersituations, the control system submits to the patient's desire. Thisdoes not mean that the control system shuts down, or does not providethe usual safety features. In fact, all of the control systems fall backon their emergency braking to keep the patient safe. When the controlsystem has had to brake to avoid an obstacle or has given up trying tolead the patient on a particular path, the patient must disengage thebrakes (via a pushbutton) or re-engage the path following (again via apushbutton) to regain control or allow collaboration again. This letsthe patient select the walker's mode manually when they disagree withthe control system's choices.

FIG. 6 shows a portable embodiment of the present invention where thevoice recognizer is housed in a wrist-watch. As shown in FIG. 6, thedevice includes a wrist-watch sized case 1380 supported on a wrist band1374. The case 1380 may be of a number of variations of shape but can beconveniently made a rectangular, approaching a box-like configuration.The wrist-band 1374 can be an expansion band or a wristwatch strap ofplastic, leather or woven material. The wrist-band 1374 further containsan antenna 1376 for transmitting or receiving radio frequency signals.The wristband 1374 and the antenna 1376 inside the band are mechanicallycoupled to the top and bottom sides of the wrist-watch housing 1380.Further, the antenna 1376 is electrically coupled to a radio frequencytransmitter and receiver for wireless communications with anothercomputer or another user. Although a wrist-band is disclosed, a numberof substitutes may be used, including a belt, a ring holder, a brace, ora bracelet, among other suitable substitutes known to one skilled in theart. The housing 1380 contains the processor and associated peripheralsto provide the human-machine interface. A display 1382 is located on thefront section of the housing 1380. A speaker 1384, a microphone 1388,and a plurality of push-button switches 1386 and 1390 are also locatedon the front section of housing 1380.

The electronic circuitry housed in the watch case 1380 detects adverseconditions such as falls or seizures. In one implementation, thecircuitry can recognize speech, namely utterances of spoken words by theuser, and converting the utterances into digital signals. The circuitryfor detecting and responding to verbal commands includes a centralprocessing unit (CPU) connected to a ROM/RAM memory via a bus. The CPUis a preferably low power 16-bit or 32-bit microprocessor and the memoryis preferably a high density, low-power RAM. The CPU is coupled via thebus to processor wake-up logic, one or more accelerometers to detectsudden movement in a patient, an ADC 102 which receives speech inputfrom the microphone. The ADC converts the analog signal produced by themicrophone into a sequence of digital values representing the amplitudeof the signal produced by the microphone at a sequence of evenly spacedtimes. The CPU is also coupled to a digital to analog (D/A) converter,which drives the speaker to communicate with the user. Speech signalsfrom the microphone are first amplified, pass through an antialiasingfilter before being sampled. The front-end processing includes anamplifier, a bandpass filter to avoid antialiasing, and ananalog-to-digital (A/D) converter or a CODEC. To minimize space, theADC, the DAC and the interface for wireless transceiver and switches maybe integrated into one integrated circuit to save space. More exemplarystructure to recognize speech is discussed in U.S. Pat. No. 6,070,140 bythe inventor of the instant invention, the content of which isincorporated by reference.

In one embodiment, the processor and transceiver conform to the Zigbeeprotocol. ZigBee was created to address the market need for acost-effective, standards-based wireless networking solution thatsupports low data-rates, low-power consumption, security, andreliability. Single chip Zigbee controllers with wireless transceiversbuilt-in include the Chipcon/Ember CC2420: Single-chip 802.15.4 radiotransceiver and the FreeScale single chip Zigbee and microcontroller. Invarious embodiments, the processor communicates with a Z axisaccelerometer measures the patient's up and down motion and/or an X andY axis accelerometer measures the patient's forward and side movements.The controllers upload the captured data when the memory is full.

The wristwatch device can also be used for home automation. The user canenjoy flexible management of lighting, heating and cooling systems fromanywhere in the home. The watch automates control of multiple homesystems to improve conservation, convenience and safety The watch cancapture highly detailed electric, water and gas utility usage data andembed intelligence to optimize consumption of natural resources. Thesystem is convenient in that it can be installed, upgraded and networkedwithout wires. The patient can receive automatic notification upondetection of unusual events in his or her home. For example, if smoke orcarbon monoxide detectors detect a problem, the wrist-watch can buzz orvibrate to alert the user and the central hub triggers selected lightsto illuminate the safest exit route.

In another embodiment, the watch serves a key fob allowing the user towirelessly unlock doors controlled by Zigbee wireless receiver. In thisembodiment, when the user is within range, the door Zigbee transceiverreceives a request to unlock the door, and the Zigbee transceiver on thedoor transmits an authentication request using suitable securitymechanism. Upon entry, the Zigbee doorlock device sneds signals to thelighting, air-conditioning and entertainment systems, among others. Thelights and temperature are automatically set to pre-programmedpreferences.

Although Zigbee is mentioned as an exemplary protocol, other protocolssuch as Bluetooth and WiFi and WiMAX can be used as well.

While the foregoing addresses the needs of the elderly, the system canassist infants as well. Much attention has been given to ways to reducea risk of dying from Sudden Infant Death Syndrome (SIDS), an afflictionwhich threatens infants who have died in their sleep for heretoforeunknown reasons. Many different explanations for this syndrome and waysto prevent the syndrome are found in the literature. It is thought thatinfants which sleep on their backs may be at risk of death because ofthe danger of formula regurgitation and liquid aspiration into thelungs. It has been thought that infants of six (6) months or less do nothave the motor skills or body muscular development to regulate movementsresponsive to correcting breathing problems that may occur during sleep.

In an exemplary system to detect and minimize SIDS problem in an infantpatient, a diaper pad is used to hold an array of integrated sensors andthe pad can be placed over a diaper, clothing, or blanket. Theintegrated sensors can provide data for measuring position, temperature,sound, vibration, movement, and optionally other physical propertiesthrough additional sensors. Each pad can have sensors that provide oneor more of the above data. The sensors can be added or removed asnecessary depending on the type of data being collected.

The sensor should be water proof and disposable. The sensor can beswitch on off locally or remotely. The sensor can be removable or clipon easily. The sensor can store or beam out information for analysispurpose, e.g. store body temperature every 5 seconds. The sensor can beturn-on for other purposed, e.g. diaper wet, it will beep and allow ababy care provider to take care of the business in time. The array ofsensors can be self selective, e.g., when one sensor can detect strongheart beat, it will turn off others to do so.

The sensor can be used for drug delivery system, e.g. when patient hasabdomen pain, soothing drug can be applied, based on the level of painthe sensor detects, different dose of drugs will be applied.

The array of sensors may allow the selection and analysis of zones ofsensors in the areas of interest such as the abdomen area. Each sensorarray has a low spatial resolution: approximately 10 cm between eachsensor. In addition to lower cost due to the low number of sensors, itis also possible to modify the data collection rate from certain sensorsthat are providing high-quality data. Other sensors may include thoseworn on the body, such as in watch bands, finger rings, or adhesivesensors, but telemetry, not wires, would be used to communicate with thecontroller.

The sensor can be passive device such as a reader, which mounted nearthe crib can active it from time to time. In any emergency situation,the sensor automatically signals a different state which the reader candetect.

The sensor can be active and powered by body motion or body heat. Thesensor can detect low battery situation and warn the user to provide areplacement battery. In one embodiment, a plurality of sensors attachedto the infant collects the vital parameters. For example, the sensorscan be attached to the infant's clothing (shirt or pant), diaper,undergarment or bed sheet, bed linen, or bed spread.

The patient may wear one or more sensors, for example devices forsensing ECG, EKG, blood pressure, sugar level, among others. In oneembodiment, the sensors are mounted on the patient's wrist (such as awristwatch sensor) and other convenient anatomical locations. Exemplarysensors include standard medical diagnostics for detecting the body'selectrical signals emanating from muscles (EMG and EOG) and brain (EEG)and cardiovascular system (ECG). Leg sensors can include piezoelectricaccelerometers designed to give qualitative assessment of limb movement.Additionally, thoracic and abdominal bands used to measure expansion andcontraction of the thorax and abdomen respectively. A small sensor canbe mounted on the subject's finger in order to detect blood-oxygenlevels and pulse rate. Additionally, a microphone can be attached tothroat and used in sleep diagnostic recordings for detecting breathingand other noise. One or more position sensors can be used for detectingorientation of body (lying on left side, right side or back) duringsleep diagnostic recordings. Each of sensors can individually transmitdata to the server 20 using wired or wireless transmission.Alternatively, all sensors can be fed through a common bus into a singletransceiver for wired or wireless transmission. The transmission can bedone using a magnetic medium such as a floppy disk or a flash memorycard, or can be done using infrared or radio network link, among others.

In one embodiment, the sensors for monitoring vital signs are enclosedin a wrist-watch sized case supported on a wrist band. The sensors canbe attached to the back of the case. For example, in one embodiment,Cygnus' AutoSensor (Redwood City, Calif.) is used as a glucose sensor. Alow electric current pulls glucose through the skin. Glucose isaccumulated in two gel collection discs in the AutoSensor. TheAutoSensor measures the glucose and a reading is displayed by the watch.

In another embodiment, EKG/ECG contact points are positioned on the backof the wrist-watch case. In yet another embodiment that providescontinuous, beat-to-beat wrist arterial pulse rate measurements, apressure sensor is housed in a casing with a ‘free-floating’ plunger asthe sensor applanates the radial artery. A strap provides a constantforce for effective applanation and ensuring the position of the sensorhousing to remain constant after any wrist movements. The change in theelectrical signals due to change in pressure is detected as a result ofthe piezoresistive nature of the sensor are then analyzed to arrive atvarious arterial pressure, systolic pressure, diastolic pressure, timeindices, and other blood pressure parameters.

While this invention has been particularly shown and described withreferences to preferred embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the spirit and scope of theinvention as defined by the appended claims.

What is claimed is:
 1. A monitoring system for a person in a building,comprising: a plurality of 802-protocol wireless bases positioned in thebuilding; a sensor mounted on the person's finger in order to detectblood-oxygen levels and pulse rate; and a wristwatch wearable devicecoupled to the sensor and having a wireless 802-protocol transceiveradapted to communicate with the wireless bases, and a recognizer in thewearable device, the recognizer including one of: a neural network,fuzzy logic, Hidden Markov Models (HMMs), dynamic time warping engine,wherein the recognizer receives data from one or more sensors anddetects a predetermined condition based on the heart signal sensor, andwherein the wearable device sends an assistance request for the personbased on the predetermined condition, and wherein the wearable deviceprovides the person with home automation control.
 2. The system of claim1, wherein the wearable device determines position based ontriangulation.
 3. The system of claim 1, wherein the wearable devicedetermines position based on RF signal strength and RF signal angle. 4.The system of claim 1, further comprising a device to detect aconfirmatory signal from the person before sending an assistancerequest.
 5. The system of claim 4, wherein the device comprises aswitch.
 6. The system of claim 1, comprising a heart rate sensor or ablood pressure sensor or an EKG sensor.
 7. The system of claim 1,comprising an EKG sensor having a plurality of EKG nodes attached to aplurality of locations on the body of the person.
 8. The system of claim1, wherein the wireless transceiver communicates over an 802 protocol.9. The system of claim 1, further comprising computer readable code totransmit a help request to a remote computer.
 10. The system of claim 1,further comprising computer readable multimedia communication code toallow a user and a remote person to audibly communicate with each other.11. The system of claim 10, wherein the remote person includes a doctor,a nurse, a medical assistant, or a caregiver.
 12. The system of claim 1,further comprising code on a computer readable medium to store andanalyze patient information.
 13. The system of claim 12, wherein thepatient information includes medicine taking habits, eating and drinkinghabits, sleeping habits, or exercise habits.
 14. The system of claim 1,wherein the wireless transceiver complies with 802.15 hardwarerequirements and uses a non-802.15 stack.
 15. The system of claim 1,comprising code on a computer readable medium to store and analyzeinformation relating to the person's ambulation.
 16. The system of claim1, comprising a global positioning system (GPS) receiver to detectmovement and where the person falls.
 17. The system of claim 1,comprising code on a computer readable medium to map the person'slocation onto an area for viewing.
 18. The system of claim 1,comprising: one or more cameras positioned to capture a video of thepatient; and a server coupled to the one or more cameras, the serverexecuting code to detect a dangerous condition for the patient based onthe video and allow a remote third party to view images of the patientwhen the dangerous condition is detected.
 19. A monitoring system for aperson in a building, comprising: a sensor mounted on the person'sfinger in order to detect blood-oxygen levels and pulse rate; an802-protocol compatible wristwatch wearable device coupled to the heartsignal sensor and having a wireless transceiver adapted to communicatewith the wireless bases and a recognizer receiving data from one or moresensors and detecting a predetermined condition, and wherein thewearable device sends an assistance request for the person based on thepredetermined condition and provides the person with home automationcontrol.
 20. The system of claim 19, wherein the wearable devicecontrols lighting, heating and cooling from anywhere in the building.